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Consider a (not necessarily irreducible) unitary representation of a semisim- 
ple Lie group G in a Hilbert space H (for instance we can consider an action 
of the group G in some function space on some homogeneous space, a tensor 
product of irreducible unitary representations, a restriction of an irreducible 
unitary representation to a subgroup etc.). There arises a natural problem: to 
decompose our representation into irreducible representations, in other words, 
to find the spectrum of the representation. Problems of this type were widely 
investigated for the last 50 years (since [Kre]) and by now many such spectral 
problems have been solved completely or partially. 

In several simple cases the spectra are continuous and more or less 'uniform'. 
But sometimes such spectra contain strange discrete increments which means 
that there exist minimal G-invariant subspaces in the Hilbert space H. First 
examples of discrete increments were observed in [Nai], [Puk], [H-Ch], [Moll] 
(1961 - 1966) and now there exists a large literature devoted to discrete spec- 
tra, see [Boy], [Ism], [Mol3-4], [Str], [Far], [Pat], [F-Jl], [F-J2], [Sch], [Kobl-2], 
[RSW], [Tsu], [How], [Ada], [Li], [BO] etc. 

Quite often 'discrete increments' are very singular ('exotic') representations 
and it is very difficult to construct such unitary representations in some other 
way. Even Harish-Chandra's discrete series still cannot be considered handy 
objects. The remarkable paper [F-Jl] (1980, see also [F-J2]) contained a seri- 
ous generalization of Harish-Chandra's discrete series and this paper started a 
systematic search of 'exotic' unitary representations in discrete spectra. 
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In [Nerl] (1986), [01s3],[01s4], [NO], [Ner2] it was observed that very often 
discrete increments to spectra are related to some functional analytic phenom- 
ena, namely to so-called 'trace theorems' (i.e theorems about restrictions of 
discontinuous functions to submanifolds) . On various analytic trace theorems 
see [RS], IX.3,IX.9, [ESh], 2.3, [Beu], [Bar], 5. 12, [NR], [Rud], [Via], [VS], [Ner2]. 

This paper is some kind of addendum to papers [NO] , [Ner2] of Olshanskii 
and myself. Its purpose is to discuss some open problems related to applications 
of trace theorems in noncommutative harmonic analysis. 

I am grateful to G. I. Olshanskii for our collaboration. I also thank V. F. 
Molchanov, H. SchlichtkruU, G. Zuckerman, B. Orsted, M. Flensted- Jensen, T. 
Kobayshi, A. G. Sergeev, G. Olafsson, B. Kostant, A. Dvorsky, R. S. Ismagilov, 
R. Howe, V. M. Gichev, V. V. Lebedev for discussions, comments and references. 
These notes are mainly based on my lectures [Ner5] delivered at the school 
'Analysis on homogeneous spaces' (Tambov, august 1996) and I also thank the 
organizers of that school. 



1 Tensor product of two representations of 
SL{2, M) of the complementary series 

1.1. Complementary series. Consider the circle with the coordinates 
z = e**^ (where \z\ = 1, e [0, 27r]). Let — 1 < s < 1, s ^ 0. Further consider the 
scalar product 



< / / |sin(0i-</.2)/2|(i+^)/2 



Jo 



in the space of C°°-functions on the circle. The functions e™"^ are pairwise 
orthogonal with respect to this scalar product and 

. >_ r(n+(l + .)/2) 

^ ' '^-r(n+(l-.)/2) 

If s > 0, then the integral (1) diverges. Nevertheless, the formula (H) makes 
sense and we can consider (2) as the definition of the divergent integral (|l]). 
Denote by Tis the completion of the space C°°(T^) with respect to the scalar 
product (|l|). We have 

/(0) = ^c„e'"* e ^ 5]nnc„|2 < oo 

i.e. Jis is a Sobolev space on the circle T^. 

We realize the group S'L(2,R) as the group of matrices g = 



a b 
b a 

satisfying the condition |a|^ — = 1. Define a representation of the group 
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S'L(2,IR.) in the space Tig by the formula 

The representations Tg are unitary in the Hilbert space Tia (this fact is simple 
but it is not trivial). They are called the complementary series representations 
of SL{2,M.) . Note that the representations Tg and T-s are equivalent. 

Consider the tensor product Tg^ (g) Tg^ of two complementary series repre- 
sentations. The space Hg^ Hg^ is a space of functions on the two-dimensional 
torus : zi = e^'^^ , Z2 = e*'^^ equipped with the scalar product 

</l,/2> 

.fl{4'l,4'2)f2i'ipl,1p2)d(f)ld(f)2d'lpidlp2 

I sin((/)i - i/'i)/2|(i+«i)/2| sin(02 - V2)/2p+«2)/2 
The group SL2 (M) acts in this space via the formula 

\ a J \bzi+ a 0Z2 + aj 

1.2. Restriction to the diagonal. 

Denote by A the diagonal (pi = (j)2 of the torus T^. 

Theorem 1.1. Let si + S2 > 1 . Then the restriction operator restricting a 
function f G C°^{T'^) to the diagonal A extends to a bounded operator R from 
the space Hsi ^Ti-g^ to the space L'^{A). 

We emphasize that the functions f G Hg^ ® Tig^ may be discontinuous and 
hence a function f & Hg has no value in a individual point of the torus. Never- 
theless, the restriction operator R is well defined. 

Proof. The space Hsi <8) 'Hs2 consists of functions 

satisfying the condition 

The restriction of / to the diagonal A has the form 

k m+n=k 

and the convergence of the series ^ Cm.n easily can be deduced from the Cauchy- 
Bunyakovsky inequality. □ 

The restriction operator is a bounded S'L(2, IR.)-interwining operator from 
Tg^ ®Ts2 to Tsi+S2_i. The representation Tg^ ^Tg^ is unitary and hence Tg^j^g^-i 
is a subrepresentation of Tg^ ® Tg^ . 
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1.3. Dual picture: distributions supported on the diagonal. Now 

let si + S2 < —1 (recall that T-s Tg). It turns out that in this case the 
Hilbert space Tis^ ® contains distributions supported on the diagonal A. 
Obviously the space of distributions supported on the diagonal is an invariant 
subspace. It is easy to show that the representation of SL{2, M) in this subspace 
is equivalent to Tg-^^g^^i and we again obtain an embedding of a complementary 
series repesentation into the tensor product. 

1.4. Separation of a discrete increment. Let si = S2 = s < —1/2. 
Consider the operator 



sm 



Then the operator J intcrwines the representation Ts®Tg and the representation 
a b \ ^ ^,azi + b az2 + b^,Y . — i-Iit: — i-i 



(To^To) - f(z,,Z2) = f{^^,^^)\bz^ + anbz2 + a 
\ " J bzi + a bz2 + a 

The last representation is the standard representation of 51/(2, R) in L^(T^). 
The kernel of J consists of distributions supported on the diagonal and the 
image is dense in L^. Hence the representation Tg (g) Tg is equivalent to the 
direct sum of the representation Ti+2s and the representation of SL{2, R) in L'^ 
on the torus. 

Remark. The operator J is a operator of multiplication on nonsmooth 
function in space of distributions. This operator is unbounded and hence it is 

ncsscsary to be careful, sec [NO], §7 for details and for the case Si 7^ S2- 

Remark. The existence of the embedding of Tsj_|_s2_i into T^^ Tg^ was 
obtained by Pukanzky [Puk] in 1961, and an analogous result for SL{2, C) was 
obtained by Naimark [Nai] (1961). The explanation of this phenomenon using 
the restriction operator wasn't observed before [Nerl] (1986). 



2 Hilbert spaces defined by positive defined ker- 
nels. 

2.1. Positive definite kernels. Let S be a set. A function K{x,y) on 
S X S is called a positive definite kernel if for each finite collection of points 
xi, . . . ,x„ G 'E we have 



det 



f K{xi,xi) K{xi,X2) 

K{x2,Xi) K{X2,X2) 
\ K{Xn,Xi) K{Xn,X2) 



K{xi,Xn) ^ 
K{x2,Xn) 



> 



To each positive definite kernel K{x,y) we associate a Hilbert space H and 
fixed system of vectors & H enumerated by a; e S such that 
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1. The linear span of the system "iix is dense in H 

2. For all x,y G E we have 

<'iy,'i'x >H== K{x,y) (3) 

The system is called a supercomplete basis or an overfull system. 

Let h e H. We consider the function fh{x) = Jh{x) on S given by formula 

JF{x)^ fh{x)^<h,^x> (4) 

where x G S. 

We will identify the space H and the image of the operator J . In other words 
we consider the Hilbert space H as a space of functions on S. 

2.2. Hilbert spaces of holomorphic functions. Let be an open 
domain in C". A function K(z,u) on x is called a reproducing kernel on SI 
if 

1. K{z,u) is a positive definite kernel 

2. The function K{z,u) is holomorphic with respect to the variable z and 
antiholomorphic with respect to the variable u. 

Consider the associated Hilbert space H and let us realize it as a space of 
functions on fl by the formula (||). It is easy to see that the functions fh are 
holomorphic, i.e. the Hilbert space H can be considered as a Hilbert space of 
holomorphic functions on f2. 

2.3. Example: Berezin spaces on Cartan domains. Let p < q. Denote 
by Bp^q the space of all p x g-matrices z satisfying the condition ||z|| < 1, i.e 
Bp^q is a Cartan domain of the first type. 

The pseudounitary group U {p, q) consists of {p + q) x {p + g)-matrices g = 

^ ^ satisfying the condition 

The group U (p, q) acts on Bp q by the fractional-linear transformations 

z ;= (a + zc)-i(6 + zd) (6) 

The stabilizer of the point z ~ consists of matrices having the form ^ ^ ^ 
where a G U{p),d G U{q). Hence Bp q is the Riemannian symmetric space 

U{p,q)/{U{p) X U{q)) 

Theorem 2.1. (see [Berl], see also [Wal]) Let s — 0, 1, 2, . . . ,p — 1 or 

s > p — 1. Then the kernel 

Ks{z,u) = det^'*(l - zu*) 
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is positive definite on Bp^q. 

Denote by Vs the Hilbert space of holomorphic functions on Bp,q defined by 
the kernel Ks{z, u) (for a discussion of this space see for instance [Berl], [NO]). 
The group U {p, q) acts on Vg by the unitary operators 



/(z) = /(zM)det-^(a + zc), 



(7) 



where z^^l is given by the formula (|6|). 

The representations are called spherical highest weight representations of 
U(p,q). 

Remark. If s is an integer, then Tg is a representation of the group U {p, q) 
itself. If s is not an integer, then the function det^'*(-) is multivalued and hence 
Tg is a representation of the universal covering group of U{p, q) (in other words, 
the representation is a projective representation of the group U{p, q) ). 

Remark. If s > p — 1 , then the space Vs contains all polynomial functions 
on the Cartan domain Bp^q. If s = 0, 1, . . . ,p — 1, then all functions f £ Vs 
satisfy the following system of partial differential equations: 



det 



dzi 



V 



d 



9 \ 



for all a = 2, 3, . . . , s + 1 and all satisfying the inequalities < < g, 

< > < p. 

2.4. Cartan domains of second and third types. The Cartan domain 
Cn of third type is the set of all n x n symmetric matrices z with complex 
coefficients satisfying the condition ||z|| < 1. The group G of biholoniorphic 
transformations (automorphisms) of the domain C„ consists of transformations 

a b 
c d 



having the form (|^) , where a matrix g = 



satisfies the condition 



a b 
c d 



1 

-1 



a b 
c d 



1 

-1 



d ~ a, c = b. 



(8) 



It is easy to see that the group G is isomorphic to the real symplectic group 
Sp(2n, R) and the domain C„ is the Riemannian symmetric space 

C„ = Sp{2n,R)/U{n). 

The kernel 

Ks{z, u) :— det~'*(l — zu) 
is positive definite if and only if 

{s = 0, 1/2, . . . , (n - l)/2} or {s > {n ~ l)/2}. 
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The Cartan domain Dn of the second type is the set of all n x n skew- 
symmetric matrices z satisfying the condition ||z|| < 1. The group G of au- 
tomorphisms of the domain Dn is the group of transformations (|^), where the 

matrix a = \ ^, I satisfies the condition 
\c d ) 

(: ;)(" ^)'^(? 

Remark. Condition (|^) implies (||). 

The group G is isomorphic to the real classical group SO*{2n) and the 
domain _D„ is the symmetric space 

Dn = SO*{2n)/U{n). 

The kernel K{z, u) — det^''(l — zu*) on Z?„ is positive definite if s = 0, 1, 2, . . . , 
n— lors>n— 1. 

Spherical highest weight representations of S'p(2n, R) and S0*{2n) act in 
Hilbert spaces associated to the kernels Ks by the formula (|^). 

2.5. Matrix-valued positive definite kernels. Consider a set S and a 
finite dimensional complex cuclidian space y. Denote by GL{y) the group of 
invertible linear operators in y. We say that a function 

L : S X S ^ GL{C,y) 

is a matrix-valued positive definite kernel (see for instance [HN]) if the hmction 

Z((z, ^); (u, ri)) :=< L{z, u)C, f] > ; (z, C), (w, »?) G S x y 

is a positive definite kernel onE x y. 

Suppose we have a matrix- valued positive defined kernel on S. Then there 
exists a unique Hilbert space H and a map 'i' : E x y H such that 

a) The map ^' is linear on each fiber z x y C 'E. x y. 

b) < 0, *(?^, ?7) >ff =< V >y 

c) The image of the map 4* is total in H. 

Again there exists a canonical map J from the Hilbert space H to the space 
of 3^-valued functions on S. Let h E H. Then the function — Jf is defined 
via the equation 

< fh{z),C >y=< K^iz,^) >H ■ 

2.6. Highest weight representations of the groups G = U{p,q), 
Sp{2n,R), S0*(2n). 

Consider the case G = U{p,q). Let p = pi (g) p2 be an irreducible finite- 
dimensional holomorphic representation of the group GL{p, C) x GL{q, C) or of 



7 



its universal covering. Further let W = Wi (g) W2 be the space of the represen- 
tation p. Assume that the function 



Kp{z, u) := pi{l — zu*) (X) ^2(1 — u*z) 



is a matrix- valued positive-definite kernel (see [01s2] for conditions of positive 
definiteness, see also [NO]). Consider the associated Hilbert space Hp of Wi (S> 
Wij-valucd holomorphic functions on -Bp,g. Then the unitary highest weight 
representation Tp of the group U(p, q) (or of its universal covering) acts on the 
space Hp by the formula 



Let G = Sp{2n,'M.) or SO*{2n). Consider a finite dimensional holomorphic 
representation p of the group GL{n,C) or of its universal covering group. As- 
sume the matrix-valued kernel Kp{z, u) = det~*(l — zu*) to be positive-definite 
on the Cartan domain C„ or £>„, respectively. Denote by Hp the associated 
Hilbert space of holomorphic vector-valued functions. Then the unitary highest 
weight representation Tp of the group G is given by the formula 



2.7. Bibliographical comments. 

a) There are also Cartan domains of 4-th type S0{n,2)/ {SO{n) x 0(2)) 
and two exceptional Cartan domains. There exist associated highest weight 
representations. The Cartan domains S0{n,2)/ {0{n) x 0{2)) are a popular 
object of complex analysis ( tube of future ), see [VS]. These domains often 
are called Lie spheres or Lie balls, but this term is a result of a philological 
mistake. (Lie sphere is a oriented sphere, oriented plane or point in M". The space 
of all Lie spheres is a homogeneous space 0(n+2)/{0{n) x 0(2)) (sec for instance 
[Cec]). The term 'Lie sphere' for a homogeneus space 0{n,2)/{SO{n) x 0(2)) 
appeared in russian translation of famous Chinese book [Hua] . It is quite strange 
to name this space by "sphere' and it was transformed to 'ball'.) 

b) Conditions for the positive-definiteness of the standard scalar- valued ker- 
nels on Cartan domains of types one through four were discovered by Berezin 
[Ber] (with some additions given by Gindikin [Gin]); Berezin and Gindikin didn't 
consider exceptional domains; later conditions for the positive-definiteness in 
the general case (including exceptional) were obtained by Wallach [Wal], Rossi 
- Vergne [RV] and Jackobsen [Jac] , proofs of these authors are essentially differ- 
ent). 

c) Conditions for the positive-definiteness of the standard kernels for Bp^g 
were obtained in [01s2] . For other domains this problem was discussed by many 
authors, the final result was obtained in [EHW]. 




(10) 
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3 Boundary values of holomorphic functions. 



3.1. L^-limits. 

Let C C" be an open domain, dH. its boundary, and be the closure of 
n. We say that f2 is a regular circle domain if 

a) for all z £ n and A G C such that |A| < 1 we have Xz € fl 

b) for all z e dft and A G C such that |A| < 1 we have Xz G il. 
Let K{z, u) be a reproducing kernel in Q satisfying the condition 

K{e"^z,e"f'u) ^ K{z,u) (11) 

and H be the Hilbert space of holomorphic functions associated to this kernel. 

Remark. Cartan domains are regular circle domains and the Berezin kernels 
satisfy the condition (p]). 

Theorem 3.1. (see [NO]) Let M C dfl be a compact subset. Let fi he a 
measure supported on M . We make the following assumptions 

a) K*{z, u) limc^i-o K{cz^ cu) exists almost everywhere on M x M with 
respect to the measure /i x /i. 

b) K* G L^{M X Af, /i X /i) and limc^i^o K {cz, cu) is dominated, i.e. there 
exists a function S{z,u) G L^{M x Af, /i x /ij such that \K{cz,cu)\ < S{z,u) 
almost everywhere on M x M . 

Then the restriction operator restricting a function f £ H to the set M is a 
well-defined operator 

H L^M, ^). 

This theorem doesn't cover all cases in which the restriction operator exists 
(see Subsections 3.2-3.3 below). Hence the following natural problem arises. 

Problem 1. Let C C" be an open domain. Let K{z, u) be a reproducing 
kernel and let H be the associated Hilbert space. Further let A^ be a submanifold 
of the Shilov boundary of fl. Find conditions for the existence of a restriction 
operator from H to some Hilbert space of functions on M. 

Remark. The case which is interesting for harmonic analysis is the follow- 
ing. Let the domain SI be a Cartan domain G/K and H — Vghe a Berezin space. 
Consider a subgroup Q C G and some Q-orbit M in the Shilov boundary of the 
domain fl. Suppose the restriction operator restricting functions / G i/s to the 
manifold M exists. Then we obtain an interwining operator from the Berezin 
space Hs to some Hilbert space H{M) of functions on M. The representation of 
Q in H{M) is equivalent to a subrepresentation of the representation of Q in H. 
If the group Q is semisimple and the orbit M is compact, then the representation 
of Q in H{M) has purely discrete spectrum. 

3.2. Restrictions of functions of polynomial grov^fth on a ball. De- 
note by Bq the unit ball 

|ziP + ... + |zj2 < 1 
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in and let H°°{Bq) be the Hardy space of bounded holomorphic functions 
on Bq. Denote by (•, •) the standard scalar product in C". 

Consider a C^-curve 'y{t) in the sphere dBq = S^'^~^. 

Theorem 3.2 (see [NR],[Rud]) Let j{t) satisfy the condition 

Vt:/m(7(0,7'(i))^0 (12) 

Then for each f G H°°{B'^) the nontangential limit f{z) as z ^ ^{t) exists 
almost everywhere on the curve ^{t). 

Remark. Obviously the vector ^'{t) is tangent to the sphere 5^^^^. Hence 
we have Re{'y{t),j'{t)) = for all t. Fix z G S'^''~^ and let u G C be a vector. 
The equation (z, v) — Q defines a subspace of codimension 1 in the tangent 
space to the sphere S'^^^^ in the point z G 5^^^^ C C". We obtain a contact 
distribution on the sphere (see [Rud], 5.4, 10.5). The condition ( [l^ ) means that 
"/(t) is transversal to this distribution. 

Denote by T)' the space of all holomorphic functions of polynomial grouth in 

feV'^3N: sup |/(z)|(l - Izp)^ < oo 

Remark. The space V is a union V — Us>ol4(Sg) of Berezin spaces 
on the ball, see Subsection 2.3. The space Vq{Bq) coincides with the Hardy 
space H^{Bq). The space Vq+i{Bq) is the Bergman space (i.e the intersection 
of L'^iBq) with the space of holomorphic functions). 

Remark. It is well known (and more or less obvious) that each function 
f € V has a limit on the boundary in the sense of distributions (see [RS],IX.3 
for a discussion of theorems of this type and further references). 

Denote by R the restriction operator resctricting a holomorphic function / 
on the ball Bq to the curve j{t). Obviously the function Rf is well-defined if 
the function / is continuous up to the boundary. 

Theorem 3.3. (see [Ner2]) Let^{t) he a C°° -smooth curve indB'' satisfying 
the condition ^1^). Then the restriction operator R restricting a holomorphic 
function to 'y{t) extends to a bounded operator from T>' to the space of distribu- 
tions on 7(i). 

3.3. Restriction of functions of polynomial growth on a polydisc. 

Denote by P" the polydisc \zi\ < l,...,|z„| < 1. Let T" be the torus 
zi = e*"^!, . . .,Zn = e*"^". Let j{t) = {4'i{t), • . • , <l>n{t)) be a time-like C°°-curve 
in T" i.e. 

Vt:0Ui) >O,...,0Ui) >0 (13) 

Denote by V the space of holomorphic functions of polynomial growth in P", 
i.e 

71 

feV'^3N: sup/(z) [](! - \^k\)^ < °o 
fe=i 
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This condition is equivalent to the existence of an L > with 
f{zi, ...,z„) = Y^ Ck^,...,kn4^ ...z^- eV and 

V- |cfci,...,fc„| 

^ [(fcl + 1) . . . (fc„ + 

Theorem 3.4. (see [Ner2]) The restriction operator R restricting a holo- 
morphic function f on P" to the curve ^{t) extends to a bounded operator from 
the space V to the space of distributions on the curve j{t). 

Example. Consider a bidisc and the Hilbert space -ffsi.sa associated to 
the reproducing kernel 

{l-ZiUiy{l-Z2U2y, 

where si,S2 > 0. It is easy to show that V = Usj^s2>o-ffsi,s2- If 7(0 has the 
form (7i(i), 72(t)) = (e**,e*'^) where a = const, then a restriction operator to 
7(f) doesn't exist for all Si,S2- If 7(f) satisfies the condition ^[{t) > 0, 72(f) < 
then the restriction operator exists only in the case Si + S2 < 1. 

3.4. A conjecture. It seems to me that Theorems 3.3-3.4 are partial cases 
of some general fact. Let f2 C be an open domain. Let M be a submanifold 
in the Shilov boundary of CI. Denote by Tm the tangent space to M in the 
point m € M. We identify Tm with a linear submanifold in C". Denote by 
Sm = i ■ Tm the linear submanifold which consists of the vectors 

i ■ {v — m) + m, 

where v £ T„i (and i'^ = —1). We say the submanifold M is time-like if for each 
point m ^ M there exists an open cone Cm C Sm with the vertex m and an 
e- neighborhood Ue{m) of m such that 

Cm no D Ue{m)n Cm- 
Conjecture. Each holomorphic function of polynomial growth in fl has a 
restriction to a time-like manifold M in the Shilov boundary in the sense of 
distributions. 

Theorems 3.3-3.4 are partial cases of this conjecture. It is also similiar to the 
standard facts on limits of functions of polynomial growth on the whole Shilov 
boundary mentioned above (see [RS], IX.3, [Via], [VS]). Neverless I couldn't 
find this fact in the literature. 



4 Positive definite kernels on Riemannian non- 
compact symmetric spaces 

4.1. Spherical kernel-representations of the groups U{p,q). Consider 
the matrix ball Bp^g, see Subsection 2.3. Let s = 0, 1, 2, . . . ,p — 1 or s > p — 1. 
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Consider the function 



Ls{z,u) = |det(l - zu*)]-^', 
where z,u e Bp q. With the notations of Subsection 2.3 we have 



Ls{z,u) = Ks{z,u)Ks{z,u) 



and hence Ls{z, u) is a positive definite kernel on Bp^q. Denote by Hs the Hilbert 
space associated with the kernel Ls(z, u). Using the formula (||) we identify this 
space with some space of real analytic functions on Bp g. 
The group C/(p, q) acts on Hs by the unitary operators 



Problem 2. Decompose the representation As 

We will call the representations Tg spherical kernel-representations of the 
group U{p, q). 

Some partial cases of this question were discussed in the end of the 70-ies, 
see [Ber2],[Rep],[Gut]. In fact, only the case where s is large has been discussed 
there. In this case the most interesting phenomena don't appear. For a while 
such problems were more or less forgotten. During the last years some of these 
problems attracted interest again (see [NO], [00], [0Z],[Dij]). 

I would like to try to explain why this problem is interesting and also to 
discuss some approaches to this problem. 

4.2. Limit as s — *■ oo. We want to show that the limit of the kernel- 
representations as s — > +00 is the canonical representation of U {p, q) in the 
space on Riemannian symmetric space U{p,q)/{U{p) x U{q)). 

Consider the system of vectors S Hs (see Subsection 2.2). Let x be a 
distribution in Bp^q with compact support. Consider the vector 0(x) G Hs 
defined by the equality 



and the scalar product {•, - j^ in the space of distributions on Bp q with compact 
support given by the formula 



We can identify the space Hs with the completion of the space of distributions 
with respect to the scalar product {•, -js- The group U{p, q) acts on this space 
of distributions by the formula 




2s 




{xi,x2}s := (e(xi),e(x2)) 

r r det(l-zz*)det(l--uw*) ^ 
Jbj,,^Jbp_^ det^(l-zM*) 



Xi{^)X2 {u)dzdzdu(lu. 



Bs{g)f{z) = f{z^^^) 
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(the formula doesn't depend on s, nevertheless, the scalar product and the 
spectrum of the representation do depend on s in an essential way). 

It appeared that for some normalizing multiplier oj{s) for all continuous 
functions 4>i(z), 4>2{z) on Bp^g with compact support we have 

lim u){s){(t)i,(f)2}s = / <j)i{z)(j)2{z)dzd:z 

4.3. Another formulation of the problem. Consider a spherical highest 
weight representation Tg of the group U{p,q) (see Subsection 2.3). Denote 

by Vs the space of the representation Tg. Denote by T* the contragradient 
representation of Tg. Consider the tensor product Tg (g) T*. This representation 
acts on the space Vg ^ Vg oi holomorphic functions on Bp^g x Bp^g via the 
operators 

{Tg ® T:)ig) = f{zr,Z2) = f{z^\zf){a + z^c)-\a + Z2c)-\ 

The U{p, (7)-invariant scalar product in the space of holomorphic functions on 
Bp^q X Bp^q is defined by the reproducing kernel 

M{zi, Z2\ wi, W2) = det~'*(l — ziu^)dct ""(l — Z2U2) 

Consider the operator 

I:Vg®Vg^Hg 

defined by the formula 

If{z) = f{z,z) 

Obviously / is a unitary operator interwining the representations 

Tg ®T*g ^Ag. 

Hence we can formulate our problem in the form: 
Problem 2'. Decompose the tensor product Tg ® T* . 

4.4. Orbits of the group U {p, q) on the Shilov boundary. 

Denote by Mp^q the Shilov boundary of -Bp,g. The elements of Mp^q are 
matrices z satisfying the condition 

z-z* = 1. 

In the other words, z is the matrix of an isometric embedding ^ C^. Hence 
Mp^q is a complex Stiefel manifold. 

The Shilov boundary of Bp^g x Bp^g is Mp^g x Mp^q. The group U{p,q) has 
(p + 1) orbits on Mp^q x Mp^g. The only invariant of an orbit is the number 

a = rk{z — u) (14) 
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i.e if (zi,Mi), (z2,U2) G Mp^q X Mp^q and rfc(zi — ui) = rk{z2 — M2), then the 
points (zi,Mi), (z2,U2) belong to the same orbit. 

We denote by Sq, the orbit corresponding to a given invariant a. 

Remark. The orbit Sq is compact, whereas the orbit Sp is open (in the 
Shilov boundary). For all a the closure of is UaKa'^a- 

4.5. Restriction of holomorphic functions to U {p,q) -orhits in the 
Shilov boundary. Fix an orbit of U{p, q) in the Shilov boundary of Bp q x 
Bp q. It can happen (and it really does happen) that for small s the function 
f G Hs = Vs ^ Vs has a well defined restriction to the orbit Sq,. In this case 
the restriction operator is an interwining operator from Vs x Vs to some Hilbert 
space of functions on . 

It can also happen (and it really does happen) that for small s all first partial 
derivatives of the function f G Hg — Vs ^ Vs have a well defined restriction to 
the orbit Sq etc. (see discussion of this phenomenon in [NO], Section 7). 

Fix s. For each a = 0, 1, . . . ,p — 1 consider the maximal number Tq, such 
that all partial derivatives of order < Tq, of functions f G Vs ® Vs have well- 
defined restrictions to S^. (these numbers aren't known, but Theorem 3.1 gives 
a possibility to estimate them; I don't know whether these estimates are best 
possible or not). If restrictions of the functions f G Hg = Vs (iS Vs to Sq, don't 
exist we suppose Tq = —1. 

Remark. For large s the restriction operators don't exist, i.e. we have 
Tq, = — 1 for all a < p — 1 (see below subsection 4.6). 

Fix a = 0, 1, ... p — 1. Consider some Iq = 1, • • • , Tq. Denote by Q[a, ia] the 
space of functions f G Vg^Vs such that all partial derivatives of / of orders < i 
equal zero on S^. We obtain a filtration 

c - 1, Tp_i] c Q[p - 1, Tp_i - 1] c . . . C Q[p - 1, 0] c 

C Q[p-2,Tp_2] C ... C Q[p-2,1] C g[p-2,0] C ... (15) 
. . . C Q[0, To] C . . . Q[0, 1] c Q[0, 0]cVs^ Vs 

Remark. For large s this filtration is trivial, but for small s it is quite long. 

Consider the representations of U {p, q) in the subquotients of this filtration. 
Obviously As = Tg ® T* is equivalent to the direct sum of the subquotients of 
the filtration (|l|). 

Remark. The representations of U {p, q) in the subquotients have simple 
interpretations. For instance {Vs ® t4)/(3[0, 0] is a subspace in the space of 
functions on the orbit Sq- The space Q[0, 1]/Q[0, 0] is a subspace in the space 
of sections of the normal bundle of the orbit Sq . The space (5[0, l]/(5[0, 2] is a 
subspace in the space of sections of the symmetric square of the normal bundle 
etc., see the discussion in [NO], §7. 

It is natural to hope that the spectrum in each subquotient is more or less 
'uniform', i.e. that the orbit structure of the Shilov boundary gives a separation 
of the quite complicated spectrum of As into different types (compare with the 
[GG]-project ). 
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4.6. Large s. If s is large enough, then the restriction operators don't exist. 
In this case the representation As is equivalent to the standard representation 
of the group U (p, q) in L"^ on the Riemannian symmetric space U{p, q) / (U (p) x 
U{q)), see [Ber2], [Rep], [Gut], [00]. A sufficient (but not nessessary) condition 
for this iss>p + q — l (i.e. T, belongs to the Harish-Chandra discrete series). 

4.6. Restriction to the compact orbit. The part of the spectrum which 
corresponds to the compact orbit Sq is purely discrete and it consists of quite 
exotic representations of U {p, q) . This part of the spectrum is not empty for 
small s (for ,s < {q — 2p — l)/4). In [NO] one can find a description of the 
analogous situation for the compact orbit of 0(j), q) in the Shilov boundary of 

In particular this gives a relatively simple way to construct singular unitary 
representations of the groups U{p, q), see [NO]. 

4.7. Spherical kernel- representations of other classical groups All 

classical Riemannian noncompact symmetric spaces G/K (up to the center of 
G) can be realized as (real, complex or quaternionic) matrix balls (see [Ner3]). 
Namely the space G/K is the space of matrices z over the field K = IR,C,EI 
(see below) satisfying the additional condition (see below) such that < 1. 

We list the symmetric spaces G/K, fields and additional conditions. 

1*. U{p, q)/{U{p) X U{q)) is the space of p x g-matrices over C. 

2* . Sp{2n,'S.)/U{n) is the space of symmetric n x n-matriccs over C. 

3*. SO*{2n)/U{n) is the space of skew symmetric n x n-matrices over C. 

4*. 0{p, q)/{0{p) X 0{q)) is the space of p x g'-matrices over M. 

5*. GL{n,'S.) / 0{n) is the space of symmetric n x n-matrices over R. 

6*. 0{n, C)/0{n) is the space of skew-symmetric n x n-matrices over R. 

7*. GL{n, C)/U{n) is the space of hermitian n x n-matrices over C. 

8*. Sp{p, q) / {Sp{p) X Sp{q)) is the space of p x g-matrices over H. 

9*. GL{n,'S) / Sp{p,q) is the space of hermitian n x n-matrices over H. 

10*. Sp{2n,C)/Sp{n) is the space of skew hermitian (i.e. z = —z*) n x n- 
matriccs over H. 

In all cases enumerated above the group G acts on the matrix ball G/K hy 
fractional-linear transformations 

z^ z^s^ — {a + zcy^b + zd) (16) 

Now let us consider positive definite kernels on G/K having the form 

Ls{z,u)= \ det{l - zu*)\-^' 

(the conditions for positivc-dcfinitcncss are different for different spaces). Con- 
sider the Hilbert space defined by the positive definite kernel Ls{z,u). We 
identify this space with a space of real analytic functions on G/K in the same 
way as in 2.1. The group G acts on Hg by the unitary operators 

As{g)f{z) = f{z^^^)\det{a + zc)\-^' 
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We call As a spherical kernel-representation of G. 
4.8. Nonspherical kernel-representations. 

Consider one of the symmetric spaces form 2*, 3*, 5*, 6*, 7*, 9*, or 10* and 
fix a finite dimensional irreducible representation p of the group GL{n, K) or of 
its universal covering. Assume that the function 



is a matrix valued positive definite kernel. Then we consider the associated 
Hilbert space Hp of real-analytic functions G/K ^ y and the unitary kernel- 
representation of G in Hp given by the formula 



Consider the cases 1*, 4*, 8* and fix a finite-dimensional irreducible represen- 
tation p = (g) p2 of the group GL{p, K) x GL{q, K) or of its universal covering. 
Assume that the function 



is a positive definite matrix- valued kernel on G/K. Then the group G acts on 
the associated space of real-analytic functions on G/K by the formula 



Remark. Our arguments from the Subsections 4.3-4.4 are valid for general 
kernel-representations. 

4.9. Another description of kernel-representations, (see [00]) For 
G = S'j3(2n, R), U{p,q), SO*{2n) a kernel-representation is a tensor product 
of an irreducible highest weight representation of G and an irreducible lowest 
weight representation of G. 

In other cases a kernel-representation of G is a restriction of a highest weight 
representation of the group G* to the symmetric subgroup G: 



4*. G = 0{p,q) G* = U{p,q) 5*. G = GL{n,R) G* = Sp{2n,R) 

6*. G = 0{n,C) G* = SO*{2n) 7*. G = GL{n,C) G* = U{n,n) 

8*. G = Sp{p,q) G* = U{2p,2q) 9*. G = GL{n,U) G* = SO*{2n) 

10*. G = Sp{2n, C) G* = Sp{4n, R) 



Remark. The cases 1* — 3* can be described in the same way. We have 
G* = G X G and the embedding G ^ G* given by the formula g i— > {g,g^), 
where is an outer automorphism of G. 

Remark. There are some additional possibilities related to highest weight 
representations of 0{p, 2) and two exceptional groups (see [00]). 

4.10. The action of the Olshanskii semigroup. For each matrix 



L{z,u) — p{l — zu*) 



Tp{g)f{z) = p{a + zc)f{z^3]) 



(17) 



Lp{z, u) := pi(l - zu*) ® /52(1 — u*z) 



Tp{g)f{z) = (pi(a + zc) ® p^id - czM))/(zM). 



ball G/K consider the set of invertible matrices g = 
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fractional-linear transformation ( p^ maps the matrix ball into itself. Obviously 
r is a semigroup and the group G is the group of invertible elements of F. The 
formula ( pT[ ) defines a representation of the semigroup T. This representation is 
irreducible and all irreducible representations of T can be obtained in this way 
(see [Olsl]). See [Olsl], [Ner3], [Ner4], Appendix A, for the explicit description 
of the semigroups T . Moreover the kernel-representations extend to represen- 
tations of some categories (see [Ner3], [Ner4], Appendix A). I don't know any 
applications of these phenomena to harmonic analysis of kernel-representations 
of groups. 

4.11. Bibliographical comments. 

a) Let T be a highest weight representation of G = S'p(2n,M), SO*{2n), 
U{p,q), . . . and S* be a lowest weight representation of G. Assume that T, 5* be 
elements of the Harish- Chandra discrete series. Then T (g) 5 is equivalent to a 
representation of G induced from an irreducible representation of the maximal 
compact subgroup K C G (see [Ber2], [Rep], [Gut]). 

b) Spherical kernel-representations for large s are equivalent to canonical 
representation of G in L\G/K) (see [00]). 

c) Discrete spectra associated to the compact orbit in the Shilov boundary 
were investigated in [NO] for the case G = 0{p,q). Analogous results are valid 
for G = U{p, q), Sp{p, q) ([01s2], [NO], 7.12.). One method of separation of the 
discrete spectrum is discussed in [Nerl], [NO], 7. 1-7. 8. I think that the restriction 
operator to the compact orbit doesn't exist for G ^ U{p, q), Sp{p, q),0{p, q). 

d) The spectra for spherical kernel-representations of U{2, 2) were obtained 
in [0Z2]. For 1 < s < 3/2 the spectrum consists of two different pieces. One 
of the pieces coincides with the spectrum of L'^ {U(2,2)/{U{2) x U{2))). The 
other piece is an integral of nontrivial representations. 

Question. It is natural to think (it is not proved) that this piece of the 
spectrum is associated to the noncompact U{2, 2)-orbit in the Shilov boundary 
of ^2,2 X -82,2- Is it true? 

If this is so, then it is the only known case where a spectrum associated to 
the noncompact orbit can be found. It is natural to think that such spectra 
exist in various spectral problems (not only for kernel-representations). 

e) The Spectrum of spherical kernel-representations of SU (1,1) was obtained 
in [VGG] and in the case of U{p, 1) it was obtained in [Dij] (the construction of 
the discrete increments was given in [Nerl]). 

f) For a Plancherel formula for tensor product of highest weight and lowest 
weight (single-valued) representations of SL{2,'U.), see [Mol5]. Berezin [Ber2] 
announced a theorem which is almost equivalent to the Plancherel formula for 
spherical kernel-representation in the case when s is large and the symmetric 
space is hermitian. Proof is published in [UU] . 

g) Nonspherical kernel-representations have discrete spectrum which is not 
associated to the compact orbit in the Shilov boundary. Some possibilities to 
find it are contained in the following two sections. A way to find the Harish- 
Chandra discrete series increments using trace theorems is proposed in [Ner2]. 
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h) Let p be the same as in 2.10. Let /9s(7) = det~^(7)p(7). Then the hmit 
of as s — > 00 is the representation of G induced from a finite dimensional 
representation of K (i.c a representation on the space of sections of a vector 
bundle on G/ K). The theory of such representations is more or less equivalent 
to Harish-Chandra's theory for L'^{G). 

e) Deformation quantization, sec [Bcr2]. As wc have seen in 4.2 Hg = Vs® Vg. 
Hence we can identify the space Hg with the space Vg of Hilbert-Schmidt 

operators. The multiplication of Hilbert-Schmidt operators induces an asso- 
ciative operation in Hs- This operation can be considered as a "deformation 
quantization", see [Ber3]. 



5 Dual pairs 

5.1. Harmonic representation of Sp{2n,M.). We recall the definition of 
the harmonic representation W2n (= Weil representation = Segal-Shale- Weil 

representation = Fricdrichs-Segal-Berezin-Shale-Weil representation = oscillator 
representation ) of the group Sp{2n,'R) (see [KV], [Ner4] for a discussion of this 
representation). 

Consider the Hilbert space of holomorphic (entire) functions on C" asso- 
ciated to the reproducing kernel cxp(^ ^j^^i) (this is the boson Fock space). Let 

€ Sp{2n, M) (sec the realization of Sp{2n, R) described in Subsec- 



tion 2.4) The harmonic representation is a unitary projective representation of 
the group Sp{2n, M) in the space F„ defined by the Berezin formula (see [Ner4]): 

^(1 (18) 

= ^^exp{(z,^) }f(^)^M-\zf)dzdzil9) 

5.2. Noncompact dual pairs. Consider the following subgroups in the 

symplectic group (noncompact Howe dual pairs): 

Sp{2k{p + q),R) D Sp{2k,R) X 0{p,q) 
Sp{2{k + l){p + q), R) D U{k, I) X U{p, q) 
Sp{Ak{p + q),M.) D S0*{2k) x Sp{p,q) 

Remark. Let us describe the first embedding. Consider the space M^" 
equipped with a nondegenerate skew-symmetric bilinear form. Consider the 
space M^'+^ equipped with a symmetric bilinear form with inertia indices p, q. 
The tensor product of symmetric and skew-symmetric forms is a skew-symmetric 
form in the space R^" (g) W'^'^ and we obtain the required embedding of groups. 
□ 
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Let us restrict the harmonic representation W2N of the respective large sym- 
plectic group to these subgroups and then let us consider the restriction to the 
subgroups 

Oip, q),Sp{2k, R), U{k, I), U{p, q), SO* {2k), Sp{p, q) 

It was proved in [How], [Ada], [Li] that the spectra of these restrictions have 
discrete increments. This construction is one of the standard ways to obtain 
singular unitary representations of the groups t/(p, q), 0{p, q), Sp{p, q). 

Proposition 5.1. (see [NO]) Each representation ofG = 0{p, q), Sp{2k, R), 
U{k,l), U{p,q), S0*{2k), Sp{p,q) which occurs in the spectrum of a dual pair 
discretely (resp. weakly) occurs in the spectrum of some kernel-representation 
discretely (resp. weakly). 

Proof. This proposition is more or less obvious. Consider for instance the 
case Sp{2k,M) x 0{p, q). The restriction of W2k(p+q) to the subgroup 

Sp{2k,R) C Sp{2k,M) X 0{j),q) C Sp{2k{p + q),R) 

is equivalent to the representation 

Wf^^^muT' (20) 

The first tensor factor is a direct sum of highest weight representations and the 
second tensor factor is a direct sum of lowest weight representations. Hence ( pO| ) 
is a direct sum of kernel-representations of the group S'p(2fc,R). 

Consider the subgroup 0{p, q) C S'p(2fc,K) x 0{p, q). Consider the following 
subgroups in Sp{2k{p + q), M): 

Sp{2k{p + q),W) D 0{p,q) x Sp{2k,R) D 0{p,q) 

II nun 

Sp{2k{p + q),R) D U{p,q) X U{k) D U{p,q) 

The restriction of W2k{p+q) to U {p, q) is a direct sum of highest weight repre- 
sentations and hence the restriction of W2k{p+q) to 0{p, q) is a direct sum of 
kernel-representations. □ 

5.3. Restriction to orbits. 

Consider the Cartan domain Cn = Sp{2N,M.)/U{N) and the reproducing 
kernel 

K{z,u) = det"^''^(l - zu*) 

on Cn. Denote by H the associated Hilbert space. The group Sp{2N,R) acts 
on H by the unitary operators 

W+,{g) = f{z^^y)det-'/'{^ + z-^) 

The representation M^27V '^^ two irreducible components of the represen- 

tation W2N (the other irreducible component can be realized in some space of 
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holomorphic vector- valued functions). Again we have a question about the re- 
strictions of holomorphic functions to 0{p, q) x Sp{2k, M)-orbits in the Shilov 
boundary of C^v- The orbit structure of the Shilov boundary in this case is very 
complicated. In any case there exists an orbit 

J^= Sp{2k,R)/U{n) X 0{p,q)/Q, 

where Q is stabilizer of a maximal isotropic subspacc in the pscudoeuclidean 
space M^'+^. I can show that the restriction operator restricting to the orbit 
exists and this observation gives a way to see a part of the discrete spectrum 
for the dual pair. It is interesting to calculate this part of the spectrum. 

Another question which seems interesting to me: is it possible to obtain 
by such way some handy realizations of some Harish-Chandra discrete series 
representations in this way ? 

6 The L^-space on Stiefel manifolds 

6.1. Stiefel manifolds. The following 10 types of homogeneous spaces 
G/Q will be called Stiefel manifolds: 

1°. 0{p,q)/0{p-t,q-s) 2°. U{p,q)/U(p-t,q-s) 
3°. Sp{p,q)/Sp{p-t,q- s) 4°. Sp{2n,m.)/Sp{2{n-t),M.) 

5°. Sp{2n,C)/Spi2{n-t),C) 6°. 0{n,C)/0{n - t,C) 
7°. SO*{2n)/SO*{2{n-t)) 

8° — 10°. The spaces of all linear embeddings 

In the last 3 cases the group G is GL{n, M.),GL{n, C),GL{n, H), respectively, 
and Q is the group of matrices having the form 

( : ) pi) 

Remark. The Stiefel manifold Sp{2n,R)/Sp{2{n - is the space of 

isometric (form preserving) embeddings of the space M^* equipped with a non- 
degenerate skew symmetric bilinear form on the space R^" equipped with a non- 
degcncratcd skew symmetric bilinear form. The other Stiefel manifolds 1° — 7° 
have an analogous description. 

6.2. Additional symmetries. 

We consider the homogeneous space G/Q = Sp{2n.M.)/ Sp{2{n — t),'R). Then 
the group Sp{2t, M) acts in the obvious way on the space of symplectic-isometric 
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embeddings M^* M^" (since it acts on the space R^*). Hence the manifold 
Sp{2n,R) / Sp{2{n - t),R) is a Sp{2t,R) x S'p(2n, R)-homogeneous space: 

Sp{2n, R) /5p(2(n-i), R) = iSp{2t, R) x Sp{2n, R))/(5p(2i, M) x 5p(2(n-i), R)) 

Analogous additional groups of symmetries exist in all the cases 1° — 10°. These 
additional symmetries are useful since the spaces L'^iG/Q) have G-spectrum 
with infinite multiplicities. 

6.3. L^-spectrum on Stiefel manifolds. Little is known about the 
spectral decomposition of L'^{G/Q). Nevertheless, this problem is interesting. 
See [Sch] and [Kobl] for Flensted- Jensen type constructions of discrete spectra 
in on 

0{p,q)/0{p~r,q), U{p,q)/U{p - r,q), Sp{p,q)/Sp{p - r,q) 

For the case r = 1 the Plancherel formula is obtained in [0Z1]. Some construc- 
tions for discrete increments in spectra of 

L\U{p, q)/{U{p -t,q-t)x U{p) X U{q))) C L^{U{p, q)/{U{p) X U{q))) 

are contained in [RSW]. 

Remark. The cases G/Q, where G = GL(n,R), GL(n,C), GL(n,H) are 
very simple. We can consider two-step unitary induction from the subgroup Q 
(see (21)) to a parabolic subgroup and from a parabolic to the group GL{n) 
itself. 

Proposition 6.1. Each representation of G luhich is contained in the spec- 
trum of L^{G/Q) discretely (resp. weakly) is contained in the spectrum of some 
kernel-representation of G discretely (resp. weakly). 

Proof. We use arguments from [How], [NO]. Consider for instance the case 
G = Sp{2n,M.). The representation 

of Sp{2n,M.) is equivalent to the representation of Sp{2n,'R) in i'^(R^"). Hence 
the representation 

{W2n «> W;j^^'' = WZ^'' ® (W^2*„)®"' 

is equivalent to the representation of Sp{2n,M.) in on the space Mat2k,2n 
of all 2k X 2n-matrices. A generic orbit of Sp{2n,M.) in Mat2k,2n is a Stiefel 
manifold Sp{2n,R) / Spi2{n - A;),R). 

Remark. For other groups G Proposition 6.1 can be proved using the same 
arguments. The basic observation is 

W2n ^ i'(K") 

GL(n,R) 
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(cf. the real model of the harmonic representation in [KV]). 

6.3. Some pseudo-Riemannian symmetric spaces. In the obvious way 
we obtain 

L^{0{p,q)/{0{p)xO{p-r,q))) C {0{p, q) /0{p - r, q)) 

L^{U{p,q)/{U{r)xU{p-r,q))) C L^U{p,q)/U{p - r,q)) 
L\Sp{p, q)/{Sp{r) x Sp{p - r, q))) C {Sp{p, q)/Sp{p - r, q)) 

and the spectra of these spaces are contained in the spectra of Stiefel manifolds. I 
don't know such embeddings of spectra for other pseudo-Riemannian symmetric 
spaces. 

7 Miscellaneous 

7.1. Molchanov representations. In 1970 Molchanov [Mol2] investigated the 
most degenerate representations of the groups 0{p,q). These representations 
are obtained by inducing from maximal parabolic subgroups and taking unitary 
subquotients of these induced representations. Molchanov representations have 
very simple K = 0{p) x 0((j')-spectra (spherical harmonics with multiplicities 
less or equal 1). For this reason Molchanov representations are very simple and 
handy objects. 

These representations are also very singular in different sences: they have 
small rank in the sence of Howe (on Howe rank see [Li]) and their matrix coef- 
fitients decrease very slowly (on decreasing of matrix elements see [CHH]). 

Recently those representations attracted interest again, see [HT]. The most 
degenerate Molchanov representations are now very popular objects, see [BK], 
[Kos], [K0] (they arc minimal representations in the sense of A. Joseph). See in 
[BK] discussion of minimal representations of 0{p, q) and some other series of of 
simple groups. The most peculiar case is the 'semi-exceptional' group 0(4, 4), 
sec [Kosj. 

I only wanted to mention that Molchanov representations have very inter- 
esting spectral properties. Restrictions of Molchanov representations to the 
subgroup 0{p,q — 1) were discussed in [Mol] and [K0]. The problem is solved 
for continuous series of Molchanov representations (in this case we have continu- 
ous spectrum with discrete increments, see [Mol2]) and for the most degenerate 
representation (in this case the spectrum is discrete, see [K0]). 

Questions. It seems to me that it is a very interesting problem to decom- 
pose the tensor product of two Molchanov representations. Another interesting 
question is 'hunting' for discrete spectra in tensor products of several Molchanov 
representations (paper [Nerl] contains a false remark on this subject). 

Remark. It is possible to show that spectra of tensor products of Molchanov 
representations contains spectra of dual pairs. I don't know relations between 
spectra of kernel-representations and spectra of tensor products of Molchanov 
representations. 
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7.2. Representations unitary with respect to indefinite scalar prod- 
uct. There exists a old folk- lor (originated by Ismagilov and Molchanov) on 

harmonic analysis of rcprcsemtations which are unitary in indefinite hcrmitian 
scalar products (partially it is published, see [Ism], [Sul]). It is known that these 
representations often (more often than on definite unitary case) contains discrete 
increments. It was observed in [Ncrl] that this discrete increments usually can 
be explained by restriction theorems. There arises very much nonsolved prob- 
lems (for instance it is possible to consider kernel-representations with small or 
negative parameter s). Neverless the main (well-known) question now is the 
following: 

Question. Let a representation p of a semisimple group G unitary in a 

indefinite hermitian metric is given. When indefinite scalar product defines by 
some natural way a topology in space of representation? 

The question is very dangerous and sophisticated. On harmonic analysis in 
handy case of Pontryagin spaces there are several publications, see for instance 
[Ism], [Sul] (a Pontryagin space is a linear space provided by nondegenerate 
hermitian form with finite negative inertia index). For some discussion of non- 
Pontraygin case (for highest weight representations) scc> [HN] . 

7.3. Harish-Chandra discrete series increments. Tools which are 
proposed in [NO] don't give possibilities to search Harish-Chandra discrete series 
increments to unitary representations. Neverless some possibilities to expose 
Harish-Chandra discrete series increments with restriction theorems exist (this 
is the subject of [Ner2]). 

7.4. On totaly discrete spectra. In [01s4], [Kob2] there were discov- 
ered large classes or restriction problems which have totaly discrete spectra. 
Relations between these works are not clear to me, probably they have open 
nondensc int(TS(x;tion. It seems to me that it is interesting to understand are 
these spectra related to some functional-theretical phenomena or not. 
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